Towards the Generation of Overspecified Multimodal Referring Expressions

Recently, there has been an increased interest in the generation of multimodal referring expressions (e.g., Salmon-Alt and Romary, 2000; Lester et al., 1999; André and Rist, 1996; Reithinger, 1992; Claassen, 1992). The task involved in generation of multimodal referring expressions is to decide what is the best way to refer to a target via different modalities in the current context. Existing algorithms that generate referring expressions focus on distinguishing references (e.g., descriptions that only apply to the referent and not to any other object in the domain) or minimal references (e.g., the shortest distinguishing descriptions possible for a given referent). However in human conversation overspecified references (e.g., descriptions which include more information than necessary for identification) are relatively more common (Arts, 2004; Beun and Cremers, 1998; Pechmann, 1989). But in fact, what kind overspecification do human speakers produce and why? And secondly, how can this be mimicked in automatic generation? This paper attempts to answer these questions by: (1) addressing overspecification as occurring in human communication with respect to two production experiments used for the evaluation of the graph-based multimodal algorithm proposed by Krahmer and van der Sluis (2003) and (2) proposing a variant of this multimodal algorithm that can generate overspecified descriptions based on strategies observed in human communication. The basic multimodal algorithm, an extension of the graphbased algorithm (Krahmer et al., 2003), differs from other multimodal algorithms in that it can generate various pointing gestures differing in precision as a function of distance, where the type of pointing gesture co-varies with the amount of linguistic information in the accompanying description. The algorithm employs a set of cost functions to select properties, relations and pointing gestures according to preference. In this paper a notion of certainty is proposed that together with the cost functions causes a flexibility with which the algorithm can generate the whole range of possible referring expressions, from minimal ones to the utmost overspecified ones. This paper is organized as follows: In Section 2 overspecification in human communication is discussed. Section 3, presents a variant of the multimodal graph-based algorithm, that is able to generate unimodal and multimodal overspecified referring expressions. Section 4 ends this paper with a discussion.